Repository navigation
phase 3: QualityPanel with runtime aggregator from per-file dataqual - #24
Merged
neuromechanist merged 2 commits intoMay 18, 2026
Conversation
Reworked Phase 3 against the current epic architecture (deferred README + BIDS tree fetching). Ships the data-quality half: pipeline success, three percent-retained histograms, line-noise distribution. Phase 3 originally expected the hallu sync to produce a pre-aggregated aggregates.json + hed-summary.json. Hallu instead emits per-file dataqual.json files at every leaf path under <id>/qa/. New lib/qa-aggregate.ts walks the data.nemar.org listing tree recursively, fetches per-file dataquals in parallel, and assembles QaAggregates at request time. [id].astro's Cache-Control absorbs the fan-out cost for repeat visits. QualityPanel renders for datasets with QA in S3 today (31 datasets synced 2026-05-18; on002718 renders 18 files end-to-end). For datasets without QA, the panel hides itself gracefully. Out of scope: - HED wordcloud (depends on hed-summary.json which hallu doesn't produce) - Demographics (depends on participants.tsv ingestion) - FileVis/FileView modals (file-level click integration; will close #21 separately) Closes #21 in part — file modals tracked as a follow-up. Refs #11 (the original Phase 3 PR; supersedes its scope half-A).
Deploying nemar-website with
|
| Latest commit: |
5891c4a
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://09f4bfb8.nemar-website.pages.dev |
| Branch Preview URL: | https://feature-issue-1-phase3-rewor.nemar-website.pages.dev |
Fixes from code-reviewer, silent-failure-hunter, and pr-test-analyzer: - Cap per-file fan-out at 25 parallel fetches (was unbounded). Large datasets with hundreds of files could saturate data.nemar.org or graze Cloudflare's worker subrequest ceiling. - Surface truncated listings: console.warn when the listing endpoint reports it served a partial subtree, instead of silently producing an under-counted aggregate. - Distinguish non-404 listing/file failures: console.warn on 5xx or parse errors so a worker-side regression is visible across all datasets instead of silently hiding QualityPanel. - Warn when more than zero per-file fetches fail; an aggregate over 20 of 100 files looks plausible but is misleading. - Stricter test router: exact-URL match, so a regression that drops the trailing-slash append is caught by the mocked tests (the bare form 308-redirects to a broken /data/ prefix in production). - New tests: session-structured (sub/ses/eeg) recursion, partial mid-walk 404, 'cleaning' branch (no ICA percent), 'other' branch (line-noise only, no percent fields). 107/107 tests pass, typecheck clean.
Contributor
Author
pr-review-toolkit review (sonnet)Three reviewers, seven findings, all addressed in 5891c4a. code-reviewer (REQUEST CHANGES)
silent-failure-hunter (2 SILENT FAILURES)
pr-test-analyzer (NEEDS MORE COVERAGE)
107/107 tests pass (was 102, +5 new), typecheck clean, biome clean. Live |
neuromechanist
merged commit May 18, 2026
f738a55
into
feature/issue-1-epic-nemar-redesign
1 check passed
This was referenced May 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reworks Phase 3 (the data-quality half) against the current epic architecture. Supersedes the scope of #11 for QualityPanel; closes the QualityPanel half of #21.
Summary
Ships the QualityPanel + 4 charts + HED wordcloud component for
/dataset/<id>pages. Renders when QA artifacts exist in S3 (today: 31 mirrored datasets synced 2026-05-18); hides cleanly when not.Why a rework instead of a rebase
Phase 3 PR #11 forked before the epic moved README + BIDS tree fetching off the page (
c15a199). A direct rebase produces unresolvable conflicts in[id].astrobecause Phase 3's commit relies on synchronous imports (fetchGithubReadme,findReadmeEntry,getManifest,buildTree) that the epic moved into/api/dataset/[id]/*endpoints.What's new
src/lib/qa.ts— types + client (from Phase 3: Data quality charts + HED wordcloud + file modals #11, unchanged), plusgetQaAggregatesnow delegates to the runtime aggregator instead of fetching a staticaggregates.jsonthat hallu doesn't producesrc/lib/qa-aggregate.ts— new. Walksdata.nemar.org/<id>/qa/recursively, fetches every*_dataqual.jsonin parallel, assemblesQaAggregates(file count, pipeline status, per-file percent arrays for goodData/goodChans/goodICA, line-noise dB). Handles both shallow (sub-XXX/eeg/) and session-aware (sub-XXX/ses-YY/eeg/) BIDS layouts.src/lib/qa-aggregate.test.ts— 4 tests (root 404, no per-file files, full aggregation, icaFail handling)src/components/QualityPanel.astro+PipelineSuccessChart,HistogramChart,AgeGenderChart,HedWordcloud— from Phase 3: Data quality charts + HED wordcloud + file modals #11src/pages/dataset/[id].astro— addsgetQaAggregates(id)to the existingPromise.all, renders<QualityPanel aggregates={qaAggregates} hedSummary={null} />between Readme and BidsTreeVerified on live data
/dataset/on002718renders QualityPanel with:Cold SSR adds ~2-2.5s (the recursive listing fan-out). The page's existing
s-maxage=300Cache-Control absorbs repeat visits.Deliberately deferred
hed-summary.jsonwhich hallu doesn't produce today; QualityPanel passeshedSummary={null}for nowdata.nemar.org/<id>/qa/sub-XX/ses-YY/<modality>/returnschildren: 0even though S3 has the files. Worker-side bug, separate from this PR; the aggregator handles it gracefully by hiding the panel. Will file against nemar-cli.Tests + lint